1.1.1 Goals of this section

• Contingency tables
  • a.k.a., crosstabs, frequency tables
  • 2-way (2 variables) and 3-way (3 variables)
  • Residuals
• Chi-square tests of independence
  • Fisher’s Exact Test
  • Conditional, marginal, Simpson’s paradox

1.1.2 Goals of this lecture

• Contingency tables
• Study design
• Measures of relationship
• Chi-square tests

2.1.1 Contingency table

• Shows the relationship between two (or more) variables
  • Frequency of responses in each category
• Do you believe in an afterlife?

2.1.2 Notation for frequencies

2.1.3 Notation for frequencies

• \(n_{11}\), \(n_{12}\), \(n_{21}\), \(n_{22}\) are joint frequencies
• \(n_{1+}\), \(n_{2+}\), \(n_{+1}\), \(n_{+2}\) are marginal frequencies

2.1.4 Notation for probabilities

2.1.5 Notation for probabilities

• \(p_{11}\), \(p_{12}\), \(p_{21}\), \(p_{22}\) are joint probabilities = \(\frac{n_{ij}}{n}\)
• \(p_{1+}\), \(p_{2+}\), \(p_{+1}\), \(p_{+2}\) are marginal probabilities = \(\frac{n_{i+}}{n}\) or \(\frac{n_{+j}}{n}\)
• Note: \(p\) used for sample values, \(\pi\) for corresponding population values

2.1.6 Marginal probability

• Marginal probability: Probability of \(X\) or \(Y\), collapsing over the other
  • What is the distribution of \(X\), ignoring \(Y\)?
  • What is the distribution of \(Y\), ignoring \(X\)?

2.1.7 Marginal probability: Do you believe in an afterlife?

• Start with frequencies

2.1.8 Marginal probability: Do you believe in an afterlife?

• Divide each value by \(n\): The total sample size

• \(\color{red}{Joint}\) probabilities sum to 1: \(\color{red}{0.452 + 0.353 + 0.103 + 0.092} = 1\)
• \(\color{blue}{Marginal}\) probabilities for rows sum to 1: \(\color{blue}{0.555 + 0.445} = 1\)
• \(\color{blue}{Marginal}\) probabilities for columns sum to 1: \(\color{blue}{0.805 + 0.195} = 1\)

2.1.9 Conditional probability

• Typically, \(X\) is an explanatory variable (predictor)
  • \(Y\) is an outcome variable
• Conditional probability: Probability of \(Y\) at a given value of \(X\)
  • When \(X = 1\), what is the distribution of \(Y\)?
  • When \(X = 2\), what is the distribution of \(Y\)?
• Conditional probability is the \(\color{red}{joint~value}\) divided by the \(\color{blue}{marginal~value}\) for that value of \(X\)
  • It is conditional on that value of \(X\)

2.1.10 Conditional probability: Do you believe in an afterlife?

2.1.11 Conditional probability: Do you believe in an afterlife?

2.2.1 True state vs test result

2.2.2 Sensitivity and specificity

• Sensitivity
  • Probability of positive for individuals who actually are positive
  • “True positive”
  • Statistical power
• Specificity
  • Probability of negative for individuals who actually are negative
  • “True negative”
  • 1 - type I error rate

2.2.3 Sensitivity and specificity

• Sensitivity = \(P(positive~test | diseased) = \frac{n_{11}}{n_{1+}} = \frac{1}{1} = 1.0\)
• Specificity = \(P(negative~test | not~diseased) = \frac{n_{22}}{n_{2+}} = \frac{87}{99} = 0.88\)

2.2.4 Sensitivity and specificity

• Ideally, both sensitivity and specificity are high
  • They are probabilities, so near 1
• Several things are related
  • Sensitivity and specificity
  • Joint probabilities / frequencies
  • Marginal distribution of the \(X\) variable (base rate)

3.1.1 Designing a study

• When designing a study
  • What do we constrain vs allow to vary?
  • What comes first?
• Design leads to analysis
  • Two groups, each with 3 time points
    • Repeated measures ANOVA or mixed model
  • Observe two continuous variables
    • Correlation or linear regression

3.1.2 Fixed vs random

• The marginal frequencies of a contingency table can be fixed or random

• Fixed: Chosen by the researcher

  • e.g., Collect data on 50 men and 50 women

• Random: Vary depending on the sample

  • e.g., Collect data on gender in the sample

• Note 1: “Fixed” and “random” are kind of (but not exactly) like “manipulated” and “measured”

  • More complicated: “fixed” isn’t always \(X\) – sometimes it’s \(Y\)

• Note 2: This is just one of many definitions of “fixed vs random”

3.1.3 Why do we care if they’re fixed or random?

• Probability = random divided by fixed
  • If there are no fixed marginals, we can’t calculate a probability
• Ratio = random divided by random
  • We can always calculate ratios (e.g., odds ratios…)
• Basically, what is “fixed” is what you can “condition on”

3.2.1 Study designs

• Three study designs with different fixed and random marginals
  • Multinomial (or cross-sectional)
  • Retrospective
  • Prospective (or product binomial)
• Design of the contingency table determines
  • What is conditioned on
  • What are probabilities
  • How you can talk about the relationship in the table

3.2.2 Overall study design

• Relationship between aspirin use (vs placebo) and heart attack
  • \(X\): Aspirin vs placebo
  • \(Y\): Heart attack vs no heart attack

3.2.3 Multinomial

• Collect data from \(n\) people
  • Measure aspirin vs placebo, heart attack vs not

3.2.4 Retrospective

• Collect data from specific numbers of heart attack and non patients
  • Measure whether they took aspirin

3.2.5 Prospective

• Collect data from specific number of aspirin and placebo people
  • Measure whether they have a heart attack

3.2.6 Other considerations

• Cross-sectional vs longitudinal
  • How long does it take to work?
  • Reverse causality
• Retrospective most useful for rare or unpredictable outcomes
  • Cancer, heart attack, extreme events
• Measures of association depend on the design
  • Some are only possible with prospective design

4.1.1 Why does design matter?

• Probability vs ratio
  • What is fixed and what is random?
• Design of study and contingency table determine
  • What is conditioned on
  • What are probabilities
  • How you can talk about the relationship in the table

4.1.2 Why does design matter?

• Three ways to talk about relationships in contingency table
  1. Difference in proportion
     • Prospective only
  2. Relative risk
     • Prospective only
  3. Odds or odds ratio
     • Any design

4.1.3 Aspirin treatment for heart disease

• Relationship between aspirin use (vs placebo) and heart attack
  • \(X\): Aspirin vs placebo
  • \(Y\): Heart attack vs no heart attack
• Prospective design: The \(X\) marginals are fixed

4.2.1 Difference in proportion: General

• Null hypothesis
  • \(H_0\): \(\pi_1 = \pi_2\) or \(\pi_1 - \pi_2 = 0\)
• Observed difference
  • \(p_1 - p_2 = \frac{n_{11}}{n_{1+}} - \frac{n_{21}}{n_{2+}}\)
• Note: Only for prospective designs (\(X\) marginals fixed)

4.2.2 Difference in proportion: Aspirin example

• Null hypothesis
  • \(H_0\): \(\pi_{yes|placebo} = \pi_{yes|aspirin}\) or \(\pi_{yes|placebo} - \pi_{yes|aspirin} = 0\)
• Observed difference
  • \(p_1 = p_{yes|placebo} = \frac{189}{11034} = 0.017\)
  • \(p_2 = p_{yes|aspirin} = \frac{104}{11037} = 0.009\)
  • \(p_1 - p_2 = 0.017 - 0.009 = 0.008\)

4.2.3 Difference in proportion inference: General

• Standard error for the difference
  • \(SE = \sqrt{\frac{p_1(1 - p_1)}{n_{1+}} + \frac{p_2(1 - p_2)}{n_{2+}}}\)
• Large sample confidence interval on the difference
  • \((p_1 - p_2) \pm z_{\alpha/2}(SE)\)

4.2.4 Difference in proportion inference: Aspirin example

• Standard error for the difference
  • \(SE = \sqrt{\frac{p_1(1 - p_1)}{n_{1+}} + \frac{p_2(1 - p_2)}{n_{2+}}} = \sqrt{\frac{0.017(0.983)}{11034} + \frac{0.009(0.991)}{11037}} = 0.0015\)
• Large sample confidence interval on the difference
  • \((p_1 - p_2) \pm z_{\alpha/2}(SE)\)
  • \((0.017 - 0.009) \pm 1.96(0.0015)\)
  • \([0.0047, 0.0107]\)

4.3.1 Relative risk: General

• Null hypothesis
  • \(H_0\): \(\frac{\pi_1}{\pi_2} = 1\)
• Observed relative risk
  • \(\frac{p_1}{p_2} = \frac{\frac{n_{11}}{n_{1+}}}{\frac{n_{21}}{n_{2+}}}\)
    
• Note: Only for prospective designs (\(X\) marginals fixed)

4.3.2 Relative risk: Aspirin example

• Null hypothesis
  • \(H_0\): \(\frac{\pi_{yes|placebo}}{\pi_{yes|aspirin}} = 1\)
• Observed relative risk
  • \(p_1 = p_{yes|placebo} = \frac{189}{11034} = 0.017\)
  • \(p_2 = p_{yes|aspirin} = \frac{104}{11037} = 0.009\)
  • \(\frac{p_1}{p_2} = \frac{0.017}{0.009} = 1.818\)

4.3.3 Relative risk inference: General

• Standard error for the natural log of the relative risk
  • \(SE = \sqrt{\frac{1}{n_{11}} + \frac{1}{n_{21}} + \frac{1}{n_{1+}} + \frac{1}{n_{2+}}}\)
• Large sample confidence interval on the relative risk
  • Calculate in terms of ln(relative risk)
    • \(ln\left(\frac{p_1}{p_2}\right) \pm z_{\alpha/2}(SE)\)
  • Then exponentiate to convert back to relative risk metric

4.3.4 Relative risk inference: Aspirin example

• Standard error for the natural log of the relative risk
  • \(SE = \sqrt{\frac{1}{n_{11}} + \frac{1}{n_{21}} + \frac{1}{n_{1+}} + \frac{1}{n_{2+}}} =\)
  • \(\sqrt{\frac{1}{189} + \frac{1}{104} + \frac{1}{11034} + \frac{1}{11037}} =\)
  • \(0.1228\)

4.3.5 Relative risk inference: Aspirin example

• Large sample confidence interval on the relative risk
  • Calculate in terms of ln(relative risk)
    • \(ln\left(\frac{p_1}{p_2}\right) \pm z_{\alpha/2}(SE)\)
    • \(ln\left(\frac{0.017}{0.009}\right) \pm 1.96(0.1228)\)
    • \([0.3569, 0.8384]\)
  • Then exponentiate to convert back to relative risk metric
    • \([e^{0.3569}, e^{0.8384}]\)
    • \([1.4289, 2.3126]\)

4.3.6 Assymetric confidence limits on relative risk

4.3.7 Assymetric confidence limits on relative risk

4.4.1 Odds

• Odds = probability of “success” divided by probability of “failure”
  • Ranges from 0 to \(+\infty\)
    • Odds > 1: success is more likely than failure
    • Odds < 1: failure is more likely than success
    • Odds = 1: success and failure are equally likely
• Odds can be used with any study design

4.4.2 Odds

• Probability of a “success” divided by probability of “not a success”
  • \(odds = \frac{p}{(1-p)}\)

4.4.3 Odds ratio

• Odds ratio is a ratio of odds: Ratio of ratios

\[\theta = \frac{odds_1}{odds_2} = \frac{p_1/(1 - p_1)}{p_2/(1 - p_2)} = \frac{n_{11}/n_{12}}{n_{21}/n_{22}} = \frac{n_{11}n_{22}}{n_{12}n_{21}}\]

• Ranges from 0 to \(+\infty\)
  • Odds ratio > 1: odds of success is more likely in group 1 than in group 2
  • Odds ratio < 1: odds of failure is more likely in group 1 than in group 2
  • Odds ratio = 1: odds of success in group 1 and group 2 are equal

4.4.4 Odds ratio: General

• Null hypothesis
  • \(H_0: \frac{\pi_1/(1 - \pi_1)}{\pi_2/(1 - \pi_2)} = 1\)
• Observed odds ratio
  • \(\theta = \frac{p_1/(1 - p_1)}{p_2/(1 - p_2)} = \frac{n_{11}/n_{12}}{n_{21}/n_{22}} = \frac{n_{11}n_{22}}{n_{12}n_{21}}\)
• Note: Odds and odds ratios can be used with any study design

4.4.5 Odds ratio: Aspirin example

• Null hypothesis
  • \(H_0\): \(\frac{\pi_{yes|placebo}/\pi_{no|placebo}}{\pi_{yes|aspirin}/\pi_{no|aspirin}} = 1\)
• Observed odds ratio
  • \(\theta = \frac{n_{11}/n_{12}}{n_{21}/n_{22}} = \frac{189 / 10845}{104 / 10933} = \frac{0.0174}{0.00951} = 1.832\)
  • Observed ln(odds ratio) = \(ln(1.832) = 0.605\)

4.4.6 Odds ratio inference: General

• Standard error for the natural log of the odds ratio
  • \(SE = \sqrt{\frac{1}{n_{11}} + \frac{1}{n_{12}} + \frac{1}{n_{21}} + \frac{1}{n_{22}}}\)
• Large sample confidence interval on natural log of the odds ratio
  • Calculate in terms of ln(odds ratio)
    • \(ln\left(\theta\right) \pm z_{\alpha/2}(SE)\)
  • Then exponentiate to convert back to odds metric

4.4.7 Odds ratio inference: Aspirin example

• Standard error for the natural log of the odds ratio
  • \(SE = \sqrt{\frac{1}{n_{11}} + \frac{1}{n_{12}} + \frac{1}{n_{21}} + \frac{1}{n_{22}}} =\)
  • \(\sqrt{\frac{1}{189} + \frac{1}{10845} + \frac{1}{104} + \frac{1}{10933}} =\)
  • \(0.1228\)

4.4.8 Odds ratio inference: Aspirin example

• Large sample confidence interval on the log of the odds ratio
  • Calculate in terms of ln(odds ratio)
    • \(ln\left(\theta\right) \pm z_{\alpha/2}(SE)\)
    • \(ln(1.832) \pm 1.96(0.1228)\)
    • \([0.3647, 0.8462]\)
  • Then exponentiate to convert back to odds ratio metric
    • \([e^{0.3647}, e^{0.8462}]\)
    • \([1.44, 2.331]\)

4.5.1 Compare: All measures

• Note: \(p_1 = n_{11}/n_{1+}\), \(p_2 = n_{21}/n_{2+}\)

4.5.2 Compare: Relative risk and odds ratio

• Relative risk = 1.818
• Odds ratio = 1.832

\[odds~ratio = \frac{p_1/(1 - p_1)}{p_2/(1 - p_2)} = relative~risk \frac{(1 - p_1)}{(1 - p_2)} \]

• When \(p_1\) and \(p_2\) are both close to 0 or both close to 1
  • Odds ratio and relative risk are very similar
  • In this example, \(p_1 = 0.017\) and \(p_2 = 0.009\)

5.1.1 Independence

• Contingency table shows the relationship between two variables
• Example
  • \(X\) = Aspirin vs placebo
  • \(Y\) = Heart attack or not
• Question: Is there a relationship between \(X\) and \(Y\)?
  • Does knowing something about \(X\) tell you something about \(Y\)?
    • Yes: \(X\) and \(Y\) are related
    • No: \(X\) and \(Y\) are independent

5.1.2 Expected value under independence

• All statistical tests involve comparing a test statistic to an expected value under the null hypothesis
  • Null hypothesis here: Independence
• Independence is related to correlation, but stronger
  • Independence between variables \(\rightarrow\) correlation = 0 \(\rightarrow\) covariance = 0
• Expected value under the null hypothesis
  • Correlation and covariance between two variables are both 0

5.1.3 Expected value of joint frequencies

• \(E(XY)\) is the joint probability distribution of \(X\) and \(Y\)
  • Cells in the table
• \(E(X)\) and \(E(Y)\) are their marginal probability distributions
  • Margins of the table

\[E(XY) = E(X) E(Y) - cov(XY)\]

• The joint frequencies depend on the marginal frequencies as well as the covariance between the variables

5.1.4 Expected value (under independence)

• If \(X\) and \(Y\) are independent, \(cov(XY) = 0\) and

\[E(XY) = E(X) E(Y)\]

• If \(X\) and \(Y\) are independent, the joint frequencies are completely determined by the marginal frequencies
  • No covariance between \(X\) and \(Y\)

5.1.5 Expected value under independence

• Expected joint frequencies: \(\mu_{ij} = \frac{n_{i+} n_{+j}}{n}\)

• Expected joint frequencies

5.1.6 Observed values: Aspirin example

5.1.7 Expected values (independence): Aspirin example

• For example: \(146.48 = \frac{11034 * 293}{22071}\)

5.2.1 Chi-square test statistic

\[\chi^2 = \sum\left(\frac{(n_{ij} - \mu_{ij})^2}{\mu_{ij}}\right)\]

• Null hypothesis
  • Observed frequencies = expected frequencies
• If all observed = expected, then \(\chi^2 = 0\)
  • As variables become more related (not independent)
    • Differences between observed and expected increase
    • \(\chi^2\) gets larger

5.2.2 Chi-square test statistic

• \(\chi^2 = \sum\left(\frac{(n_{ij} - \mu_{ij})^2}{\mu_{ij}}\right) =\)

• \(\frac{(189 - 146.48)^2}{146.48} + \frac{(10845 - 10887.52)^2}{10887.52} + \frac{(104 - 146.52)^2}{146.52} + \frac{(10933 - 10890.48)^2}{10890.48} =\)

• \(12.343 + 0.166 + 12.339 + 0.166 = 25.014\)

5.2.3 Chi-square test inference

• Null hypothesis: Independence
  • Observed frequencies = expected frequencies
• Degrees of freedom = \((I - 1)(J - 1) = (2 - 1)(2 - 1) = 1\)
  • \(\chi^2_{critical}(1) = 3.86\)
• \(25.014 > 3.86\)
  • Reject \(H_0\) that the variables are independent
    • Placebo vs aspirin is related to heart attack status

5.3.1 Assumptions of chi-square test

• All expected joint frequencies are at least 5
  • Often incorrect stated as all observed joint frequencies
    • OK to have observed cells < 5
    • Just can’t have expected values < 5
• Chi-square is a large sample test
  • \(\chi^2\) distribution is continuous
    • Sampling distribution of the test statistic doesn’t start looking like \(\chi^2\) until the sample is quite large

5.3.2 Chi-square tests are everywhere

• Some are easy to see how they relate to the \((O - E)^2 / E\) format here
  • SEM tests of model fit compare the observed covariance matrix to the expected covariance matrix
• Some are less obvious
  • Likelihood ratio test
  • \(\chi^2\) tests of regression coefficients

5.3.3 Chi-square test vs odds ratio

• \(\chi^2\) test and odds ratio
  • Two different ways of assessing the relationship between 2 variables
• \(\chi^2\) tells you about statistical significance
• Odds ratio tells you about effect size
• Use both to get a complete picture of the relationship

5.4.1 Fisher’s Exact Test

• “Lady tasting tea” problem from Fisher
  • Can the lady tell if tea or milk was put in first?
  • Very small expected values

5.4.2 Fisher’s Exact Test

• Exact test: Not an approximation via the \(\chi^2\) distribution
  • All the possible ways that observations can be distributed in cells
  • Is observed way less likely than we would expect if variables are independent?
• “Small sample test” for 2 × 2 contingency tables
  • Often used as alternative when cells < 5
  • But there are additional assumptions

5.4.3 Exact test

• Flip a coin 2 times
  • First flip: Head or tail
  • Second flip: Head or tail
  • Both flips: HH, HT, TH, TT
• Probability of HH is 0.25: 1 option out of 4 equally likely options

5.4.4 Fisher’s Exact Test

• All marginal frequencies are fixed
  • Multinomial: Total fixed
  • Retrospective: Column (outcome) marginals fixed
  • Prospective: Row (predictor) marginals fixed
• When all marginal frequencies fixed
  • Once you know 1 joint frequency (one cell)
  • You know all the other joint frequencies
  • Test statistic = \(n_{11}\)

5.4.5 Fisher’s Exact Test

6.1.1 Summary of this week

• Contingency tables
  • What they are and how they work
• Study design
  • Relates to contingency table
  • Relates to measures of relationship
• Chi-square tests
  • Fisher’s Exact Test

6.1.2 Next week

• More complicated contingency tables
  • \(2 \times 3\) (and larger) tables
  • 3-way tables: \(2 \times 2 \times 2\) tables
• Chi-square tests for these table
  • Probing the tables
  • Residuals